Many tasks in Natural Language Processing (NLP) require large amounts of annotated or transcribed language data. It is expensive to create and transcribe this data by hand. Costs can be significantly reduced through the use of crowdsourcing to accomplish simple tasks. These cost reductions often come at the price of introducing noise and consequently lowering data quality. The challenge of crowdsourcing is therefore to efficiently filter out the noise introduced by unreliable workers in order to maintain high data quality. These and other drawbacks exist with obtaining large amounts of annotated or transcribed language data for NLP.